Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses/headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for drawing or inputting text. The user's handwriting is interpreted using a handwriting recognition system or method. Other systems for handwriting input to computing devices include electronic or digital pens which interact with paper, encoded surfaces or digitizing surfaces in order to have their movement relative to the surface tracked by a computing device, such as the systems provided by Anoto AB., Leapfrog Enterprises, Inc., and Livescribe, Inc.
Regardless of the input method used, handwriting recognition systems and methods typically involve determining the initiation of a digital ink stroke, such as when first contact with a touch sensitive surface is made (pen-down event); the termination of the stroke, such as when contact with the touch sensitive surface is ceased (pen-up event); and any movement (gestures or strokes) made between stroke initiation and termination. These determined strokes are processed to recognize and interpret the input. The type of computing device or input surface can also determine the type of handwriting recognition system or method utilized. For instance, if the input surface is large enough (such as a tablet), the user can handwrite input anywhere on or above the input surface, as if the user was writing on paper. This however adds complexity to the recognition task, because the separate elements to be recognized may be related dependent of the relative positions of the elements or may be unrelated independent of their relative positions.
For example, for structured content, such as mathematical equations, tables and matrices, the relative positioning of the handwritten elements are necessary for defining the structure. Some systems are available for dealing with the recognition of mathematical matrices, for example, U.S. Pat. Nos. 7,447,360 and 8,121,412. These systems rely on indicative elements for recognition, such as brackets or spatial alignment, e.g., within rows and columns, and as such merely recognize the structure without regard to the content itself. Whilst such recognition is applicable to relatively simple and well-formed matrices, they are unable to deal with more complex matrices, e.g., containing complex elements like equations, sub-matrices, etc., ill-aligned matrix elements or matrices having empty element cells, e.g., row and column positions. Further, the described systems of these patents provide absolute recognition of these structures thereby influencing recognition of the content itself.
What is required is a system that recognizes matrices and like complex content structures, that do not rely on the input of specific designation elements or gestures and do not significantly increase processing time or complexity to the recognition of themselves whilst retaining sufficient recognition accuracy.